As I just had the need to split up two huge mbox files by date and did not find a tool that does that easily, I thought I share what I used.
Careful, it isn’t very clever, so if you use it: It comes with NO WARRANTY at all. If it eats your mail, you lost. Not my fault, you have been warned.
The task it should solve reads
- split files of size 1.8Gb and 3.2Gb, all together nearly a million mails in them,
- do not use lots of memory for this,
- no need to interactively do anything,
- split based on Date: header.
There are various ways to do it, one would be to use mutt, with something similar to the following in a muttrc file somewhere
unset confirmcreate
set delete=yes
set folder=/dev/null
and then a lot of lines looking similar to the following, just replacing the dates;
folder-hook . 'push "T~d01/03/2008-31/03/2008\n;s2008-03.mbox\n$%q"'
If you then start mutt using
mutt -F theabovemuttrc -f themboxfiletosplit it will go and split it into the YEAR-MONTH based files. I got this way from
Gerfried Fuchs, as I am no mutt user and wouldn’t have found the way to do it with mutt myself.
Drawback: Mutt needs exact date entries in the push lines. Means you have to write
29/02 if you mean “last day of Februar”, a plain stupid
31/02 will be converted to
02/03. And I was way too lazy to list all the dates for the timeframe those mboxes cover.
So, other solution. Ruby to the rescue! The following code does split the mbox into the files I want, based on the date header:
#!/usr/bin/ruby -w
# Split a mbox file into $year-$month files
# Copyright (C) 2008 Joerg Jaspert
# BSD style license, on Debian see /usr/share/common-licenses/BSD
require 'pathname'
require 'rmail'
count = 0
File.open(Pathname.new(ARGV[0]), 'r') do mbox
RMail::Mailbox.parse_mbox(mbox) do raw
count += 1
print "# count mails\n"
begin
File.open(RMail::Parser.read(raw).header.date.strftime("split/mail-%y%m"), 'a') do out
out.print(raw)
end
rescue NoMethodError
print "Couldn't parse date header, ignoring broken spam mail\n"
end
end
end
This will go through the whole mbox and store them in split/mail-$year$ month files. It has a few things you should keep in mind, if you really want to use it:
- Spam mails often have a very broken Date header. Sometimes broken enough that it can’t be parsed. I discard the “can’t be parsed” ones, as I havent seen a valid mail with that. You might want to store it in a different file, so change the part within rescue.
- Again, Spam mails often have a very broken Date header. So you will get files for years way in the past or future. In my case I could simply delete all of them, as all of those had been spam, and I knew in which timeframe the good mails got into the mbox.
The above script took, niced down with nice -15 on an otherwise used server, about 25 minutes to split around one million mails from the above mentioned large files, with no noticable memory usage (only one mail at a time in memory).
Comments: 5
Last week I was in Extreamdura, Spain, polishing fai for Lenny. So I only fixed one RC bug (494243), which was affecting Lenny the day mrvn pointed me to it, but wasnt anymore when I uploaded it yesterday. The reason is simple: quite some removals of buggy packages have taken place, yay! (And gerris was one of them...) 
For quite a while I'm already looking forward to this year's 
